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Gamification refers to transforming the environment to become more 
game-like to produce a positive experience. In this study, the researchers 
developed a gamification model, namely the GAMEBC model, to drive 
behavior change through a health awareness campaign in defeating the 
coronavirus disease 2019 (COVID-19) pandemic. The GAMEBC model was 
developed based on the self-determination theory (SDT) and gamification 
design literature. The GAMEBC model in this study involves four elements: 
competence, relatedness, autonomy, and engagement. Each element includes 
criteria that drive behavior change in health awareness campaigns. However, 
studies that validated the gamification model elements are limited, 
specifically to drive behavior change. Therefore, the content validity of the 
GAMEBC model instrument was carried out, and the analysis was based on 
thirteen expert reviews. The mean value and inter-rater agreement approach 
were implemented to examine the content validity ratio (CVR), item content 
validity index (I-CVD, and scale content validity index (S-CVI). The expert 
evaluation approach was implemented to review the GAMEBC model in 
terms of relevance and clarity. The data were analyzed using descriptive 
analysis. As a result of this work, we formulated an instrument that can be 
used to model and measure behavior change through the gamification 
approach in health awareness campaigns. 
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1. INTRODUCTION 


Validity and verification involve a critical process to produce reliable and valid instruments. 
Validity means the instrument’s capability to measure what is supposed to measure [1]-[3] whereby at the 
end of process validity, researchers can answer the research question such as “How well does the instrument 
measure what it purposes to do”. Researchers must also measure the instruments through the reliability 
process, whether consistent or stable. There are three common validation categories: criterion-related 
validity, content validity, and construct validity [4]. As content validity is a requirement for other validities, it 
is commonly reported during the instrument development process and receives the highest attention 


[3], [5], [6]. 


In addition, content validity is defined as the items’ ability to reflect the measurement elements. It 
also refers to how the instrument’s items are sampled appropriately from the specified content [7], [8]. 
Content validity is a crucial process [9] that can support abstract concepts to be observable and quantifiable 
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indices [10], and the process is studied superficially. Reliability measurement is impossible if the instrument 
lacks content validity [11], [12]. Hence, the current study is intended to establish the GAMEBC model 
instruments’ content validity as guidelines to develop and evaluate behavior change through health awareness 
campaigns to defeat the coronavirus disease 2019 (COVID-19) pandemic. The content validity of the 
GAMEBC model is continued work after the proposed model in a previous study [13]. 


2. BACKGROUND STUDY 
2.1. Proposed model 

Self-determination theory (SDT) [14], [15] defines how people can be motivated in any task that 
they want to explore. The concept of SDT has been implemented in various domains, including health [16], 
education [17], [18], and business [19], [20]. It is an important concept that supports people’s ability to 
accomplish their life. Based on SDT, humans need an element of competence, relatedness, and autonomy to 
have optimal motivation. This study proposed a new model, namely the GAMEBC model, adapted from the 
SDT model of health behavior change [21]. In this model, we also proposed a new element, engagement, that 
describes how humans need engagement to drive behavior change as a value of user experience. 


2.2. Measurement instrument validation 

A new instrument must be recognized to certify its acceptance [22]. The acceptance of instruments 
can be achieved by ensuring the reliability and validity of instruments. An instrument should be free from 
biasness to avoid inaccurate findings [23]. An instrument’s reliability is determined based on similarity with 
findings attained each repeated process [24]. The validity of instruments refers to how well the instrument 
measures what it is supposed to measure [24], [25]. Table 1 shows the proposed elements and criteria of the 
GAMEBC model. 


Table 1. Proposed elements and criteria of GAMEBC model 


Element Code Criteria 
Autonomy AUI I can create a customization profile 
AU2 I can choose the activity in the game 
AU3 Ican control interactions in the game 
AU4 Ican control interface game 
AUS I have a sense of control over the game 
AU6 I freely play the game the way I want 
Competence Col I have a different level of challenge 
C02 I have received feedback on the progress 
CO3 I can refer to the performance bar 
CO4 I have a badge as a reward for achievements 
COS I can get the point as a reward system 
CO6 I can refer to the leader board 
Relatedness RE1 Ican involve in social interactions 
RE2 I can share the information through a social interaction 
RE3 Ican join the game community 
RE4 I have experienced various types of social interactions 
RES I feel cooperative toward other users 
RE6 I can collaborate with other users 
Engagement EN1 I feel a state of curiosity 
EN2 I feel a state of flow in gameplay 
EN3 I feel intense sensations of success 
EN4 I understand the game content 
EN5 I feel a positive emotion in a learning experience 
EN6 The game is compliant with treatments 


2.3. Expert review validation and inclusion criteria 

The study aimed to develop an instrument that was valid and reliable. A descriptive analysis was 
implemented based on expert review validity to evaluate the GAMEBC model. Expert review validity is a 
content validity procedure determined by evaluating the instrument’s contents [26]. Experts’ choice depends 
on the conceptual model produced [27] and their expertise [28]. In this study, the criteria for selecting the 
expert review were based on the experts’ publications, presentations, and research experience in the areas of 
interest. For example, the current study’s main objective was to evaluate a gamification model to drive 
behavior change in defeating the COVID-19 pandemic. Therefore, the panel of experts should be familiar 
with gamification, game design, user experience, and usability. The panel of experts will evaluate each item 
and the whole proposed model. 
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The experts’ panel consisted of nine specialized academic game researchers and four 
game-industry-related professionals in this present study. Ten female and three male experts were involved. 
The experts were recruited based on their expertise in game design in terms of experience, knowledge, and 
publication. All involved experts had at least ten years of experience related to computer science. This 
study’s expert review consisted of associate professors, senior lecturers, and game industry panel members 
(project managers, assistant producers, and game developers). The experts among academicians were also 
required to fulfill four criteria, they must: i) be a full-time lecturer, ii) have at least a minimum of five-year 
teaching experience, iii) have at least a minimum of one year of experience in designing and developing 
games, and iv) have a qualification that the Ministry of Higher Education recognizes. 


3. METHOD 
3.1. Instrument development 

The GAMEBC model was developed in four primary phases. The first phase focused on formulating 
each element’s conceptual model and operational definitions to ensure its content validity. The next phase 
was where the item pool was formulated. An appropriate scale was developed in the third phase. Finally, in 
the last phase, the expert review of the instrument was conducted. 


3.1.1. Content validity 

Content validity is a process to measure whether the items are relevant, precise, and correctly 
worded. The experts were asked to respond autonomously to a questionnaire to evaluate the instrument in 
this current study. They had to rate on a four-point rating scale of each item, which is for Relevant element 
(1 = not relevant, 2 = item needs some revision, 3 = relevant but needs some minor revision, 4 = very 
relevant) and for Clarity (1 = not clear, 2 = item needs some revision, 3 = clear but needs some minor 
revision, and 4= very clear) [27]-[30]. 

Besides, the content validity evaluates the items to ensure the instruments represent the contents or 
behavior of the addressed domain. Based on the experts’ feedback, the instrument was revised and modified 
accordingly for the next phase. Thirteen experts were invited to choose the most relevant and clear item in 
the instrument, which was quantified by the content validity ratio (CVR) as in (1) and content validity index 
(CVI) as in (2). The CVI score was obtained from the total sum of the agreement scores of all items on the 
experts’ scale of ‘3’ or ‘4’. CVI approach is commonly reported for content validity in instrument 
development and can be calculated using the Item-CVI (I-CVI) and the Scale-level-CVI (S-CVI) [30]. The 
values range I-CVI is from 0 to 1 where I-CVI> 0.79, the item is relevant, between 0.70 and 0.79, the item 
needs revisions, and if the value is below 0.70, the item is eliminated [30]. 


CVR = (Ne - 2)/() (1) 
CVI = (~) (2) 


Where, Ne is the number of experts giving a rating of ‘3’ or ‘4’, N is the total number of experts. 

Moreover, S-CVI refers to the total content-validated content [30], [31], formulated through two 
methods. The first method is the general agreement among the experts (S-CVI/UA) [30], [31], and the second 
method is the average of the item-level CVIs (S-CVI/AV). The instrument would be qualified as content 
valid for the study if S-CVI/UA reaches 80% or a better agreement among experts [27], [32], [33], and if 
S-CVI/AV is greater than 0.90 [29], [34]. The S-CVI/AV is reported to be a less conservative method [30]. 
The S-CVI/UA score was generated by totaling up all items with I-CVI equal to | divided by the total 
number of items. Meanwhile, S-CVI/AV is generated by summing the I-CVIs and dividing them by the total 
number of items. S-CVI/UA = 0.8 and S-CVI/AV = 0.9 have excellent content validity [35]. 

Another category of empirical analysis was CVR, which measures the essentiality of an item [36]. 
CVR diverges between | and —1, and a higher score indicates more significant agreement among the panel 
members [30]. The minimum value of CVR to indicate the acceptable item is based on the total number of 
experts in the Lawshe table [37], [38] as shown in Table 2. In this study, the minimum value of CVR is 0.54. 

Moreover, Wynd et al. [10] recommended a Kappa formula (3) in addition to CVI be computed 
[10], although CVI is widely utilized to measure content validity. Kappa offers the degree of agreement 
beyond chance, computed using the formula (3). Kappa values above 0.74 are regarded as excellent. Between 
0.60 to 0.74 is considered good, and 0.40 to 0.59 is fair. 


K =(I — CVI — Pc) / (I — Po) (3) 
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N! 


' N 
Ne! N-A * 05 


Where, Pc = [ 


Where, Pc is the probability of chance agreement, N is number of experts, and Ne is number of experts 
agreeing on a rating of 3 or 4. 


Table 2. The lawshe table for minimum values of CVR [38] 
No. of Expert 5 6 7 8 9 10 11 12 13 14 15 20 25 30 35 40 
Min Value 0.99 0.99 0.99 0.75 0.78 0.62 0.59 0.56 0.54 0.51 0.49 0.42 0.37 0.33 0.31 0.29 


4. RESULTS AND ANALYSIS 
4.1. Content validity of the autonomy element 

Table 3 depicts the finding of the content validity of the autonomy element. The expert review panel 
regarded the items AU1, AU2, AU3, and AUS as having excellent validity (very relevance and very clear), 
with I-CVI values ranging from 0.92 to 1.00. The S-CVI/AV for the autonomy element in terms of relevancy 
and clarity was 0.91 and 0.88. Four out of six items in this element showed excellent validity, which I-CVI 
and Kappa were above 0.90, and two items were in good validity, which is I-CVI, and Kappa scored less than 
0. 78 regarding relevance and clarity. The average CVR value in the autonomy element regarding relevancy 
and clarity was 0.82 and 0.77, respectively, which means they were above the CVR’s minimum value. Items 
AU4 and AU6 had good validity but needed revision in relevance and clarity with an I-CVI value lower than 
0.80. All experts agreed that AU1, AU2, and AU3 items were excellent validity and very relevant 
(I-CVI=1.00), and all experts also agreed that the AU2 item was very clear with an I-CVI value was 1.00. 
Based on Table 3, AU4 and AU6 items needed some revision, such as rephrasing and restructuring, based on 
the panel of experts’ feedback. 


Table 3. Content validity of the autonomy element 
Relevance of the item 


Code N Ne CVR‘ I-CVP Pe Kappa‘ Evaluation 
AUI 13 13 1.00 1.00 0.000 1.00 eee 
AU2 13 13 1.00 1.00 0.000 1.00 eek 
AU3 13 13 1.00 1.00 0.000 1.00 tee 
AU4 13 10 0.54 0.77 0.035 0.76 wee 
AUS 13 12 0.85 0.92 0.002 0.92 tek 
AU6 13 10 0.54 0.77 0.035 0.76 sek 
S-CVI/AV 0.91 
Clarity of the item 
Code N Ne CVR‘ I-CVP Pe Kappa‘ Evaluation® 
AUI 13 12 0.85 0.92 0.002 0.92 tee 
AU2 13 13 1.00 1.00 0.000 1.00 eee 
AU3 13 12 0.85 0.92 0.002 0.92 tek 
AU4 13 10 0.54 0.77 0.035 0.76 wok 
AUS 13 12 0.85 0.92 0.002 0.92 tek 
AU6 13 10 0.54 0.77 0.035 0.76 sek 
S-CVI/AV 0.88 


“Content validity ratio (CVR) = (Ne — D/S 

Content validity of individual item (I-CVI)=number of experts providing a rating of 3 or 4/number of experts 
fa ota, Ae ~ = At N 

Probability of chance occurrence (Pc) = la eae *0.5 

“Kappa (K) = (I — CVI — Pc) /(I — Pc) 

“Evaluation criteria for the level of content validity: the relationship between I-CVI and Kappa; excellent 


validity=I-CVI20.78 and Kappa>0.74 (****); good validity=I-CVI<0.78 and=0.60 and Kappas0.74 (***); 
fair validity=I-CVI<0.6 and =0.40 and Kappas<0.59 (**); and poor validity=I-CVI<0.4 and Kappa<0.40 (*) 


4.2. Content validity of the competence element 

All items in the competence element in Table 4, CO1, CO2, CO3, CO4, CO5, and CO6, were 
considered very relevant and very clear (excellent validity) I-CVI and Kappa values were more than 0.80. All 
thirteen experts agreed with CO4 and CO5 items with CVR, I-CVI, and Kappa value obtained a full score of 
1.00 in terms of relevancy. Meanwhile, all experts agreed that CO3 and CO6 were very clear items with 
excellent Kappa value by scoring 1.00. The S-CVI/AV for competence element was 0.92, which had 
excellent content validity. Overall, all the items in the competence element were remained based on the score 
values rated by the experts. 
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Table 4. Content validity of the competence element 


o 


Relevance of the item 


Code N Ne CVR* I-CVP Pcf Kappa‘ Evaluation® 
CO1 13 11 0.69 0.85 0.010 0.84 aE 
C02 13 12 0.85 0.92 0.002 0.92 ETER 
CO3 13 12 0.85 0.92 0.002 0.92 ee 
CO4 13 13 1.00 1.00 0.000 1.00 KRR 
co5 13 13 1.00 1.00 0.000 1.00 aa 
C06 13 11 0.69 0.85 0.010 0.84 SHE 
S-CVIAV 0.92 
Clarity of the item 
Code N Ne CVR’ I-CVP Pcf Kappa‘ Evaluation® 
col 13 12 0.85 0.92 0.002 0.92 EARE 
C02 13 12 0.85 0.92 0.002 0.92 Ee 
CO3 13 13 1.00 1.00 0.000 1.00 KEER 
Co4 13 12 0.85 0.92 0.002 0.92 Fk 
CO5 13 12 0.85 0.92 0.002 0.92 HEE 
C06 13 13 1.00 1.00 0.000 1.00 Ee 
S-CVI/AV 0.95 


“Content validity ratio (CVR) = (Ne — %)/() 


Content validity of individual item (I-CVI)=number of experts providing a rating of 3 or 4/number of experts 


“Probability of chance occurrence (Pc) = | 


“Kappa (K) = (I — CVI — Pc) /(I — Pc) 
‘Evaluation criteria for the level of content validity: the relationship between I-CVI and Kappa; excellent 


validity=I-CVI20.78 and Kappa>0.74 (****); good validity=I-CVI<0.78 and =0.60 and Kappas<0.74 (***); 
fair validity=I-CVI<0.6 and=0.40 and Kappas0.59 (**); and poor validity=I-CVI<0.4 and Kappa<0.40 (*) 


N! 


! N 
Ne! waa *0.5 


4.3. Content validity of the relatedness element 

Based on Table 5, all items in relatedness elements were measured relevant and clear (excellent 
validity) with Kappa value scores ranging from 0.84 to 1.00. Besides, all feedback depicted excellent values 
about relevance and clarity. All experts agreed the RE1 item was very relevant and very clear with CVR, 
I-CVI, and Kappa value had a score of 1.00. All six items needed no modification and revision. 


Table 5. Content validity of the element relatedness 


Relevance of the item 


Code N Ne CVR" L-CVP PE Kappa’ Evaluation 
RE1 13 13 1.00 1.00 0.000 1.00 oka 
RE2 13 12 0.85 0.92 0.002 0.92 sess 
RE3 13 11 0.69 0.85 0.010 0.84 eka 
RE4 13 12 0.85 0.92 0.002 0.92 werk 
RES 13 11 0.69 0.85 0.010 0.84 oka 
RE6 13 11 0.69 0.85 0.010 0.84 eka 
S-CVI/AV 0.90 
Clarity of the item 
Code N Ne CVR’ I-CVP Pcf Kappa‘ Evaluation® 
REI 13 13 1.00 1.00 0.000 1.00 eka 
RE2 13 12 0.85 0.92 0.002 0.92 kaka 
RE3 13 12 0.85 0.92 0.002 0.92 eka 
RE4 13 12 0.85 0.92 0.002 0.92 esses 
RES 13 12 0.85 0.92 0.002 0.92 yess 
RE6 13 11 0.69 0.85 0.010 0.84 oka 
S-CVI/AV 0.92 


“Content validity ratio CVR = (Ne — *) / © 


Content validity of individual item (I-CVI)=number of experts providing a rating of 3 or 4/number of experts 
“Probability of chance occurrence (Pc) = [| 


— ts ose 
Ne! (N-A)! 


“Kappa (K) = (I — CVI — Pc) / (I — Pc) 


“Evaluation criteria for the level of content validity: the relationship between I-CVI and Kappa; excellent 
validity=I-CVI20.78 and Kappa>0.74 (****); good validity=I-CVI<0.78 and =0.60 and Kappas<0.74 (***); 
fair validity=I-CVI<0.6 and =0.40 and Kappas0.59 (**); and poor validity=I-CVI<0.4 and Kappa<0.40 (*) 


4.4. Content validity of the engagement element 

Table 6 shows the engagement element considered excellent in relevancy with all items EN1, EN2, 
EN3, EN4, ENS, and EN6 scored above 0.80 for the I-CVI value. However, EN2 showed good validity in 
terms of clarity with I-CVI and Kappa value less than 0.8. None of the engagement items was rated below 
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0.50 for the I-CVI value, indicating the item’s elimination. EN2 item in terms of clarity had rated a good 
validity with a score of less than 0.80. The item needed revision and modification. 


Table 6. Content validity of the engagement element 
Relevance of the item 


Code N Ne CVR" I-CVP Pcf Kappa‘ Evaluation® 
EN1 13 13 1.00 1.00 0.000 1.00 TETE 
EN2 13 12 0.85 0.92 0.002 0.92 FERN 
EN3 13 11 0.69 0.85 0.010 0.84 TAES 
EN4 13 12 0.85 0.92 0.002 0.92 FERE 
EN5 13 11 0.69 0.85 0.010 0.84 vba 
EN6 13 12 0.85 0.92 0.002 0.92 TEk 
S-CVI/AV 0.91 
Clarity of the item 
Code N Ne CVR" I-CVP Pe Kappa‘ Evaluation® 
EN1 13 12 0.85 0.92 0.002 0.92 TAER 
EN2 13 10 0.54 0.77 0.035 0.77 Fra 
EN3 13 13 1.00 1.00 0.000 1.00 AA 
EN4 13 11 0.69 0.85 0.010 0.85 EE 
ENS 13 12 0.85 0.92 0.002 0.92 MALE 
EN6 13 11 0.69 0.85 0.010 0.85 TARE 
S-CVI/AV 0.88 


“Content validity ratio CVR = (Ne — 5 / 6) 
Content validity of individual item (I-CVI)=number of experts providing a rating of 3 or 4/number of experts 


N! N 
Ne! weal * 0.5 


“Probability of chance occurrence (Pc) = |: 
“Kappa (K) = (I — CVI — Pc) / (I — Pc) 
‘Evaluation criteria for the level of content validity: the relationship between I-CVI and Kappa; excellent 
validity=I-CVI=0.78 and Kappa>0.74 (****); good validity=I-CVI<0.78 and =0.60 and Kappas<0.74 (***); fair 


validity=I-CVI<0.6 and =0.40 and Kappas0.59 (**); and poor validity=I-CVI<0.4 and Kappa<0.40(*) 


5. DISCUSSION 

The present paper evaluated the content validity of the GAMEBC Model instruments as guidelines 
to develop and evaluate behavior change through health awareness campaigns to defeat the 
COVID-19 pandemic. Thirteen experts returned their responses out of the 16 experts who initially agreed to 
participate, resulting in a return rate of 81.25%. The average S-CVI/AV in relevancy and clarity generated 
four elements resulting in 24 content validity indices, 0.91. All S-CVI/AV values in each item ranged from 
0.88 to 1.00. The results showed that the instruments reviewed by these experts fit the purpose of 
the study. Three items, which were AU4, AU6, and EN2, needed some revision in terms of relevancy and 
clarity of the instrument. Subsequently, the GAMEBC instruments were considered to contain excellent 
content validity. 


6. CONCLUSION 

The purpose of this study was to ascertain the content validity of the GAMEBC model instruments 
that were specially developed as guidelines to develop and evaluate behavior change through health 
awareness campaigns to defeat the COVID-19 pandemic. The content validity process implemented 
on the GAMEBC model was essential to produce a reliable and valid instrument. The instrument had 
excellent content validity with an average S-CVI/AV=0.91 about relevancy and clarity. The findings 
indicated that the GAMEBC model instruments developed in this study were valid, relevant, and clear. 
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